Method and apparatus for frame coding in vertical raster scan order for hevc

ABSTRACT

A method and apparatus for frame coding in adaptive raster scan order. The method includes encoding at least one of image or video utilizing input frames and at least one of a data related to the input frame to produce bitstream with raster scan order information and displacement information for producing compressed video bitstream, at decoding time, decoding at least one of the encoded bitstream with raster scan order information and displacement information for producing compressed video bitstream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/185,401, filed Nov. 9, 2018, which is a continuation of Ser. No.15/583,460, filed May 1, 2017, now U.S. Pat. No. 10,165,276, which is acontinuation of Ser. No. 14/561,816, filed Dec. 5, 2014, now U.S. Pat.No. 9,641,856, which is a continuation of Ser. No. 13/250,806, filedSep. 30, 2011, now U.S. Pat. No. 8,934,729, which claims the benefit ofU.S. Provisional Patent Application Ser. No. 61/388,478 filed Sep. 30,2010, and 61/474,435, filed Apr. 12, 2011, all of which are incorporatedherein by reference in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention generally relate to a method andapparatus for frame coding in vertical raster scan for HEVC.

Description of the Related Art

HEVC is a video coding standard being standardized to improve codingefficiency by 50% over H264/AVC. To achieve this goal, lots of newcoding tools have been proposed for HEVC. However, all the proposedcoding tools assume the horizontal raster scan of micro blocks, whichhas two problems: 1) Horizontal raster scan is not always effective inprediction especially for intra coding and 2) with sliding window memoryfor motion estimation (ME) and motion compensation (MC), large on-chipmemory is required to compensate for large vertical motion. This isespecially a challenging for UHD (ultra high definition) videos, suchas, 4k×2k and 8k×4k. While UHD video coding is one of the applicationareas of HEVC, the frame coding in horizontal scan order is not costeffective at all for UHD videos with high vertical motion.

Therefore, there is a need for a method and/or apparatus for framecoding for HEVC.

SUMMARY OF THE INVENTION

Embodiments of the present invention relate to a method and apparatusfor frame coding in adaptive raster scan order. The method includesencoding at least one of image or video utilizing input frames and atleast one of a data related to the input frame to produce bitstream withraster scan order information and displacement information for producingcompressed video bitstream, at decoding time, decoding at least one ofthe encoded bitstream with raster scan order information anddisplacement information for producing compressed video bitstream.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is an embodiment of a frame coding in horizontal raster scanorder;

FIG. 2 is an embodiment of a frame coding in vertical raster scan order;

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order;

FIG. 4 is an embodiment of a horizontal sliding window scheme forvertical raster coding order;

FIG. 5 is an embodiment depicting horizontal bottom-right to left;

FIG. 6 is an embodiment of a vertical right-bottom to top;

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor;

FIG. 8 is an embodiment of a video stabilizer in accordance with theprior art; and

FIG. 9 is an embodiment of an improved video stabilizer.

DETAILED DESCRIPTION

The proposed method and apparatus add a vertical raster scan order tothe HEVC standard and adaptively select the best coding scan orderbetween the horizontal and vertical raster scan orders based on anycriteria, such as, size of horizontal/vertical motion, codingefficiency, on-chip memory saving and etc.

The vertical raster scan is effective for the frames which have lots ofvertical discontinuities. FIG. 1 is an embodiment of a frame coding inhorizontal raster scan order. FIG. 2 is an embodiment of a frame codingin vertical raster scan order. In FIG. 1 and FIG. 2, the frame withvertical discontinuities with horizontal and vertical raster scan ordersare shown, respectively. When we apply the horizontal raster scan order,as shown in FIG. 1, more bits are used to encode micro block mode. Thisis due to incorrect mode prediction caused by discontinuity. However,the number of bits is reduced when we apply the vertical raster scanorder, as shown in FIG. 2. Hence, the coding gain will be moresignificant for intra frames. The result indicates that the verticalraster scan can reduce bitrates up to 4% for some sequences, evenwithout adaptive horizontal/vertical raster scan order decision. Thus,adaptive decision will result in even higher bit-rate saving.

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order. If a video contains large verticalmotion, the vertical sliding window scheme may not cover the largevertical motion. In this scenario, inter mode may not be used becauserequired reference area is not available. This eventually causesencoding efficiency degradation. However, this problem could be solvedby employing a horizontal sliding window scheme with vertical rastercoding order, as illustrated in FIG. 4. FIG. 4 is an embodiment of ahorizontal sliding window scheme for vertical raster coding order.Utilizing adaptively choosing raster scan order improves the codingefficiency of inter frames at the same on-chip memory requirement. Onthe other hand, the required on-chip memory size without coding gainloss is reduced.

For horizontal and vertical search range srX*srY, on-chip memory size(byte) for sliding window for 8-bit luma can be calculated as follows:

MemSizeHorOrder=picWidth*(2*srY+N),

MemSizeVertOrder=picHeight*(2*srX+N).

MemSizeHorOrder and MemSizeVertOrder are on-chip memory sizes forvertical sliding window (horizontal raster coding order) and horizontalsliding window (vertical raster coding order), respectively. N*N is thelargest coding unit, and picWidth and picHeight are the horizontal andvertical size of the picture. Based on the equations, Table 1 lists thesearch ranges for different available on-chip memory sizes for 4k×2k(3840×2160) videos. For a given on-chip memory size, srY is alwayslimited in vertical sliding window, which potentially causes encodingefficiency degradation for large vertical motion videos as illustratedin FIG. 3. By introducing vertical coding order with horizontal slidingwindow, we can remove the limitation on srY and avoid the degradationfor large vertical motion videos.

TABLE 1 Available on-chip memory sizes vs. search ranges 500 KBytes 750Kbytes 1,000 KBytes srX srY srX srY srX srY Vertical sliding unlimited+/−33 unlimited +/−65 Unlimited +/−98 window Horizontal +/−83 unlimited+/−141 Unlimited +/−199 unlimited sliding window

The idea of vertical raster scan can be realized with frame rotation.Hence, similar effect may be seen by rotating input frames withhorizontal raster scan. This information can be simply added as SEI(Supplemental Enhancement information) or VUI (Video UsabilityInformation) in bitstreams. Moreover, we can extend this idea to anyarbitrary raster scan order. We can apply one of 8 different raster canorders for each frames; horizontal top-left to right, as shown in FIG.1, horizontal top-right to left, horizontal bottom-left to right,horizontal bottom-right to left, as shown in FIG. 5, vertical left-topto bottom FIG. 2, vertical left-bottom to bottom, vertical right-top tobottom and vertical right-bottom to top FIG. 6. It also can be realizedby modifying input frames, i.e., e.g. rotating and/or flipping using SEIor VUI.

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor. The motion sensor consists one of the following sensorsor a combination of them: accelerometer, gyroscope, magnetosensor etc.The motion sensor provided the camera orientation or displacementinformation. The displacement can be specified in terms of a rotationmatrix, quarternion, euler angle etc.

FIG. 8 and FIG. 9 show a video stabilization where camera displacementinformation can be used. Video stabilization compensates for jitter invideo due to changing camera position in order to provide a steadyvideo. FIG. 8 is an embodiment of a video stabilizer in accordance withthe prior art. In FIG. 8, the video stabilization is carried out indecoder. The camera displacement parameters are deduced from decodedframes and are used to translates/rotate/warp the decoded frame togenerate stabilized video frames. Calculation of camera displacementparameters from decoded frames is very computationally intensive andprone to incorrect estimation.

On the other hand, FIG. 9 is an embodiment of an improved videostabilizer. The camera displacement information is calculated in thetransmitter by using motion sensors. The displacement information isencapsulated in SEI/VUI and transmitted inside compressed bitstreams toreceiver. The decoder at the receiver decodes the compressed bitstreamwith camera displacement SEI/VUI and generates decoded frames anddisplacement information which is then passed on to the videostabilization module which carries out video stabilization.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method comprising: determining, by atransmitter, displacement information of an image; and transmitting, bythe transmitter, the image and the displacement information in abitstream.
 2. The method of claim 1, wherein: the displacementinformation is based on motion sensor data.
 3. The method of claim 2,wherein: the motion sensor data is collected from one of anaccelerometer, a gyroscope, and a magneto sensor.
 4. The method of claim1, wherein: the transmitter includes an encoder configured to encode thedisplacement information in the bitstream.
 5. The method of claim 1,wherein: a stability of the image is based on the displacementinformation.
 6. The method of claim 1, further comprising: determining,by the transmitter, a modification of a frame of the image based on oneof a size of a horizontal motion, a size of a vertical motion, a codingefficiency, and an on-chip memory; and applying, by the transmitter, themodification to the frame.
 7. The method of claim 6, wherein: themodification includes one of a rotation or a flip of the frame.
 8. Themethod of claim 1, further comprising: determining, by the transmitter,an adaptive scan order based on one of a size of a horizontal motion, asize of a vertical motion, a coding efficiency, and an on-chip memory;and applying, by the transmitter, the adaptive scan order to a frame ofthe image.
 9. The method of claim 8, wherein: the adaptive scan orderfor the frame includes one of: horizontal top-left to right; horizontaltop-right to left; horizontal bottom-left to right; horizontalbottom-right to left; vertical left-top to bottom; vertical top-left tobottom; vertical top-right to bottom; vertical bottom-left to top; andvertical bottom-right to top.
 10. The method of claim 9, furthercomprising: transmitting, by the transmitter, the adaptive scan order.11. A non-transitory computer readable medium comprising instructionsthat, when executed by a transmitter, cause the transmitter to:determine displacement information of an image; and transmit the imageand the displacement information in a bitstream.
 12. The instructions ofclaim 11, wherein: the displacement information is based on motionsensor data.
 13. The instructions of claim 12, wherein: the motionsensor data is collected from one of an accelerometer, a gyroscope, anda magneto sensor.
 14. The instructions of claim 11, wherein: thetransmitter includes an encoder configured to encode the displacementinformation in the bitstream.
 15. The instructions of claim 11, wherein:a stability of the image is based on the displacement information. 16.The instructions of claim 11, further cause the transmitter to:determine a modification of a frame of the image based on one of a sizeof a horizontal motion, a size of a vertical motion, a codingefficiency, and an on-chip memory; and apply the modification to theframe.
 17. The instructions of claim 16, wherein: the modificationincludes one of a rotation or a flip of the frame.
 18. The instructionsof claim 11, further cause the transmitter to: determine an adaptivescan order based on one of a size of a horizontal motion, a size of avertical motion, a coding efficiency, and an on-chip memory; and applythe adaptive scan order to a frame of the image.
 19. The instructions ofclaim 18, wherein: the adaptive scan order for the frame includes oneof: horizontal top-left to right; horizontal top-right to left;horizontal bottom-left to right; horizontal bottom-right to left;vertical left-top to bottom; vertical top-left to bottom; verticaltop-right to bottom; vertical bottom-left to top; and verticalbottom-right to top.
 20. The instructions of claim 19, further cause thetransmitter to: transmit the adaptive scan order.